Large-scale Multi-modal Pre-trained Models: A Comprehensive Survey

Abstract With the urgent demand for generalized deep models, many pre-trained big models are proposed, such as bidirectional encoder representations (BERT), vision transformer (ViT), generative transformers (GPT), etc. Inspired by success of these in single domains (like computer and natural language processing), multi-modal have also drawn more attention recent years. In this work, we give a comprehensive survey hope paper could provide new insights helps fresh researchers to track most cutting-edge works. Specifically, firstly introduce background pre-training reviewing conventional learning, works process, vision, speech. Then, task definition, key challenges, advantages (MM-PTMs), discuss MM-PTMs with focus on data, objectives, network architectures, knowledge enhanced pre-training. After that, downstream tasks used validation large-scale MM-PTMs, including generative, classification, regression tasks. We visualization analysis model parameters results representative Finally, point out possible research directions topic that may benefit future addition, maintain continuously updated list models: https://github.com/wangxiao5791509/MultiModal_BigModels_Survey .